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Abstract 

Introduction: The aims of this study were to assess agreement between the currently used 2002 American- 
European Consensus Group (AECG) classification criteria and the new 2012 American College of Rheumatology 
(ACR) criteria for Sjogren's syndrome (SS) and to identify potential sources of disagreement. 

Methods: We studied 105 patients between 2006 and 2013 from the Brittany cohort of patients with suspected SS. 
AECG criteria were applied using only Schimer's test and unstimulated whole salivary flow (UWSF) to assess 
objective ocular and oral involvement, since these are the tests most physicians use in clinical practice. Agreement 
between the two sets of criteria was assessed using Cohen's k coefficient. 

Results: Of those studied, 42 patients fulfilled AECG and 35 ACR criteria. Agreement between the two sets was 
moderate (k = 0.53). Patients fulfilling ACR but not AECG criteria (n = 8) were significantly younger and had shorter 
symptom durations, but only three of them had SS in the opinion of the evaluating physician. Xerostomia and 
xerophthalmia (AECG set only) did not discriminate between patients with and without SS. The use of UWSF in the 
AECG but not the ACR criteria explained part of the disagreement. The serological item in the ACR set (positive 
rheumatoid factor and antinuclear antibody > 1:320 or anti-SSA/SSB positivity) did not result in classification 
differences compared to anti-SSA/SSB antibody alone (AECG set). Agreement between ocular staining score >3 
(ACR set) and Schirmer's test <5 mm/5 min (AECG set) was very low (k = 0.14). 

Conclusions: Agreement was only moderate between ACR and AECG criteria, suggesting these two sets would not 
select comparable patient populations. An international consensus about which classification criteria should be 
used in clinical studies is needed. 



Introduction 

The classification criteria for Sjogren's syndrome (SS) issued 
in 2002 by the American-European Consensus Group 
(AECG) have been widely used in clinical studies over the 
last decade [1]. In 2012, the Sjogren's International Colla- 
borative Clinical Alliance issued new classification criteria, 
which have been endorsed by the American College of 
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Rheumatology (ACR) [2], These new criteria are intended 
for use in patients referred to specialists because of signs or 
symptoms suggesting SS. They were developed by asking 
experts in rheumatology, ophthalmology and oral medicine 
to select the items they felt were most relevant. 

The new criteria differ substantially from the 2002 
AECG criteria in three ways: they include no subjective 
ocular and oral symptoms and no functional or morpho- 
logical tests for the salivary glands; they use a new ocular 
staining score (OSS) [3] as the only criterion for ocular 
involvement; and they allow the use of an antinuclear 
antibody (ANA) titer > 1:320 plus rheumatoid factor (RF) 
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positivity as an alternative to anti-SSA/SSB antibody 
positivity for the assessment of systemic autoimmunity. 

Here, our objectives were to evaluate agreement be- 
tween the two criteria sets and to identify sources of 
disagreement. 

Methods 

Study population 

We studied the single-centre Brittany cohort of patients 
with suspected SS included between November 2006 
and March 2013 in Brittany, France. The inclusion cri- 
teria are described elsewhere [4]. Briefly, patients were 
addressed to our consultation by their family physician, 
rheumatologist, internist, dentist or ophthalmologist if 
SS was suspected due to sicca complaints, major salivary 
gland swelling, suggestive extraglandular features or sug- 
gestive autoantibodies. Written consent was obtained 
from all participants, and the study was approved by the 
local ethics committee (Brest University Hospital). 

This study included 105 patients of the cohort who 
had all the tests available to apply both ACR and AECG 
criteria, including 99 (94.3%) women. Mean age was 
57.2 ± 13.7 years and mean symptom duration was 6.7 ± 
6.1 years. 

Clinical evaluation and laboratory tests 

Schirmer s test was considered abnormal if <5 mm/5 mi- 
nute and the unstimulated whole salivary flow if < 0.1 ml/ 
minute. ANAs were assessed on HEp-2 cells and anti-SSA 
and anti-SSB antibodies using commercial enzyme-linked 
immunosorbent assays, and RF (IgM and IgA isotypes) 
using in-house enzyme-linked immunosorbent assays. 
Minor labial salivary gland biopsy was performed in all pa- 
tients and graded according to the semi-quantitative score 
of Chisholm and Mason [5]. Salivary gland biopsy grades 
3 and 4 indicating focus scores >l/4 mm 2 were considered 
abnormal. 

Ophthalmologic evaluation 

All patients underwent a slit-lamp examination by an oph- 
thalmologist experienced in dry-eye diseases. The corneal 
fluorescein pattern was graded from 0 (no punctate epi- 
thelial erosions) to 3 (severe keratitis). A drop of lissamine 
green dye was then instilled into the inferior conjunctival 
fornix of each eye, the patient was asked to blink several 
times, and the nasal and temporal bulbar conjunctivae 
were then immediately graded semi-quantitatively from 
0 (no staining) to 3 (diffuse dot staining or confluent 
staining). 

The Sjogren's International Collaborative Clinical Alli- 
ance OSS [3] was published during the inclusion period 
for our cohort. We retrospectively computed the OSS of 
both eyes as the 0 to 12 sum of the 0 to 3 lissamine green 
scores for the nasal and temporal conjunctiva (range, 0 to 



6) plus the 0 to 3 fluorescein score for each cornea multi- 
plied by 2 (because the specific patterns of corneal fluores- 
cein staining giving additional points in the OSS (central 
staining, filaments or confluent staining) were not de- 
scribed in our protocol). OSS >3 in at least one eye was 
considered abnormal. 

Case ascertainment 

Table 1 presents the rules for classification using the 
AECG and ACR criteria sets. AECG criteria were applied 
using only Schirmer s test and unstimulated whole salivary 
flow, which are the tests we use in clinical practice to as- 
sess objective ocular and salivary gland involvement. 

The evaluating physician was also asked to define the 
most probable diagnosis in his opinion for each patient, 
without referring to specific classification criteria. All cases 
were reviewed by a panel of three experts (VD-P, AS and 
SJ-J) to reach consensus. The most probable diagnoses in 
the opinion of the experts were SS in 47 (44.8%) patients, 
idiopathic sicca syndrome in 37 (35.2%) patients, other 
connective tissue diseases in 11 (10.5%) patients and drug- 
induced sicca syndrome in 10 (9.5%) patients. 

This clinical definition of SS cases was not used as a 
gold standard to compare the diagnostic performance of 
AECG criteria and ACR criteria in terms of sensitivity 
and specificity, since the physician most probably used, 
even subconsciously, current validated classification cri- 
teria (that is, AECG criteria) to perform his diagnosis, 
leading to circular reasoning that would have overesti- 
mated AECG performance to the detriment of ACR 
criteria. 

Statistical analysis 

Statistical tests were performed using the Statistical Pack- 
age for the Social Sciences (SPSS 18.0, 2009; SPSS Inc., 
Chicago, IL, USA). Quantitative variables were described 
as mean ± standard deviation and qualitative variables as 
number (%). Agreement between classification criteria sets 
and between criteria was evaluated using Cohens kappa 
coefficient (k). To compare patient groups, we used the 
Mann- Whitney test, Fishers exact test or the chi-square 
test as appropriate. 

Results 

Classification criteria 

Of the 105 patients, 42 (40.0%) fulfilled the AECG cri- 
teria and 35 (33.3%) fulfilled the ACR criteria (Table 2). 
Agreement between the two criteria sets was moderate 
(k = 0.53). Patients fulfilling only the ACR criteria were 
significantly younger and had shorter symptom dura- 
tions than did patients fulfilling the AECG criteria 
(mean age, 46.6 ± 15.8 vs. 60.0 ±11.4 years; and mean 
symptom duration, 2.9 ± 2.4 vs. 7.7 ± 6.8 years; P < 0.001 
for both comparisons). 
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Table 1 Pragmatic AECG [1] and ACR [2] classification criteria for Sjogren's syndrome 





Pragmatic 2002 AECG criteria 


2012 ACR criteria 


Items 


1. Ocular dryness symptoms 


1. Positive anti-SSA or anti-SSB antibodies or 
positive rheumatoid factor plus ANA > 1:320 




2. Oral dryness symptoms 


2. Focus score >1 focus/4 mm 2 on minor 
salivary gland biopsy 




3. Ocular signs: Schirmer's test <5 mm/5 minutes 


3. Keratoconjunctivitis sicca with ocular 
staining score >3 




4. Focus score >1 focus/4 mm 2 on minor salivary gland biopsy 






5. Salivary gland involvement: unstimulated whole salivary flow <0.1 ml/minute 






6. Positive anti-SSA or anti-SSB antibodies 




Rules for 
classification 


Presence of any four of the six items with at least item 4 or 6, or presence of any three 
of the four objective items (items 3, 4, 5 and 6) 


In a patient with suspected Sjogren's 
syndrome, any two of the three items 



Exclusion criteria for both criteria sets are head-and-neck radiation, graft-versus-host disease, hepatitis C infection, acquired immunodeficiency syndrome or 
sarcoidosis. Pre-existing lymphoma and use of anticholinergic drugs are exclusion criteria only in the AECG criteria, whereas amyloidosis and lgG4-related disease 
are exclusion criteria only in the ACR criteria. 

We did not use in this study the whole set of AECG criteria, but only the tests we use in clinical practice (pragmatic AECG criteria); that is, only the unstimulated 
whole salivary flow to assess salivary gland involvement (and not salivary scintigraphy or parotid sialography which are considered obsolete by most physicians), 
and only the Schirmer's test to assess ocular signs (and not vital dye staining graded according to van Bijsterveld method). ACR, American College of 
Rheumatology; AECG, American-European Consensus Group. 



Description of the patients with discordant classification flow or Schirmer's test, but seven had abnormal salivary 
Table 3 details the features of the eight patients fulfilling gland biopsy. Only three of them had SS according to 
ACR criteria but not AECG criteria and of the 15 patients the physicians opinion, and other more likely diagnoses 
fulfilling AECG criteria but not ACR criteria. All patients were rheumatoid arthritis (n = 2), systemic lupus erythe- 
fulfilling only AECG criteria had sicca complaints, either matosus (n = 1), undifferentiated connective tissue dis- 
abnormal unstimulated whole salivary flow or Schirmer's ease (n = 1) and idiopathic sicca syndrome (n = 1). 
test, and either anti-SSA/SSB antibodies or abnormal sal- 
ivary gland biopsy. Five of them had extraglandular in- Subjective sicca complaints (AECG set only) 
volvement, and all of them had SS in the opinion of the The subjective sicca symptoms (xerophthalmia and xer- 
evaluating physician. ostomia) were noted in nearly all of the patients (re- 
Among the eight patients fulfilling only ACR criteria, spectively in 92.4% and 94.3%), suggesting no ability of 
none of them had abnormal unstimulated whole salivary these symptoms to discriminate between patients with 

Table 2 Characteristics of patients meeting ACR and/or AECG criteria for Sjogren's syndrome 

Patients fulfilling 





Both sets 


ACR set only 


AECG set only 


Neither set 




(n = 27) 


(n = 8) 


(n = 15) 


(n = 55) 


Age (years) 


60.7 ± 1 1 .7 


46.6 ± 15.8 


59.3 ± 11.1 


56.4 ±14.5 


Symptom duration (years) 


9.5 ± 7.4 


2.9 ± 2.4 


4.7 ± 4.3 


6.4 ±5.7 


Female 


25 (92.6) 


8 (100) 


14 (93.3) 


52 (94.5) 


Xerophthalmia 


26 (96.3) 


6 (75.0) 


14 (93.3) 


50 (90.9) 


Xerostomia 


27 (100) 


6 (75.0) 


15 (100) 


51 (92.7) 


Salivary flow <0.1 ml/minute 


18 (66.7) 


0(0) 


10 (66.7) 


16 (29.1) 


Schirmer's test <5 mm/5 minutes 


22 (81.5) 


1 (12.5) 


12 (80.0) 


13 (23.6) 


OSS >3 


24 (88.9) 


6 (75.0) 


0(0) 


19 (34.5) 


Anti-SSA or anti-SSB positivity 


18 (66.7) 


4 (50.0) 


4 (26.7) 


1 (1.8) 


ANA >1:320 


25 (92.6) 


5 (62.5) 


8 (53.3) 


18 (32.7) 


RF positivity 


14(51.9) 


4 (50.0) 


1 (6.7) 


6 (10.9) 


Anti-SSA/SSB or RF plus ANA > 1:320 


21 (77.8) 


4 (50.0) 


4 (26.7) 


1 (1.8) 


Focus score >1 


24 (88.9) 


7 (87.5) 


11 (73.3) 


6 (10.9) 



Data presented as mean ± standard deviation or number (%). Xerophthalmia and xerostomia referred to subjective complaints from the patients. ACR, American 
College of Rheumatology; AECG, American-European Consensus Group; OSS, ocular staining score; ANA, antinuclear antibody; RF, rheumatoid factor. 



Table 3 Description of the patients fulfilling only ACR criteria or only AECG criteria 


Patient 


Sex 


Age 
(years) 


Duration 

(years) 3 


Eye b 


Mouth c 


Arthritis d 


Lung 8 Neuro f 


Parotid 9 


UWSF h 


Schirmer 1 


OSS J 


Chisholm k 


ANA 


Anti- 
SSA 1 


Anti- 
SSB 1 


RF (n< 
0.2) 


ACPA (n 

< 20) 


igG 
(mg/l) 


Clinicar 


Detailed characteristics of the eight patients fulfilling only ACR criteria 
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Detailed characteristics of the 15 patients fulfilling only AECG criteria 
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ACPA, anti-citrullinated peptide antibodies; ACR, American College of Rheumatology; AECG, American-European Consensus Group; ANA, antinuclear antibody; F, female; IgG, serum level of immunoglobulin G; M, male; 
OSS, ocular staining score; SS, Sjogren's syndrome. RA, rheumatoid arthritis; RF, rheumatoid factor; sicca, idiopathic sicca syndrome; SLE, systemic lupus erythematosus; UCTD, undifferentiated connective tissue disease; 
UWSF, unstimulated whole salivary flow. a Duration of the symptoms. Xerophthalmia (0, absent; 1, present). c Xerostomia (0, absent; 1, present). d Presence of swollen and tender joints on the day of examination, 
includes the presence of chronic dry cough or computed tomography-proven interstitial lung disease. f Patient with objective peripheral neuropathy. 9 Presence or history of parotidomegaly. h Abnormal if <0.1 ml/minute. 
'Schirmer's test abnormal if < 5 mm/5 minutes. J Abnormal if > 3. k Chisholm's grading of the lymphocytic infiltration on minor salivary gland biopsy (grades 3 and 4 correspond to a focus score >1 focus of more than 50 
lymphocytes/4 mm 2 ). 'Anti-SSA and anti-SSB antibodies (n < 40). ""Diagnosis made by the physician, without reference to specific classification criteria. 
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and without SS in this population. The proportions of 
patients with xerophthalmia or xerostomia were not sig- 
nificantly different in the group fulfilling both criteria 
sets and in the group fulfilling neither criteria set (P = 
0.35 and P = 0.2, respectively). 

Functional salivary-gland assessment (AECG set only) 

Salivary flow was decreased in 66.7% of patients fulfilling 
both criteria sets and in 29.4% of those fulfilling neither 
criteria set (P = 0.001). No patient fulfilling only the 
ACR set had a decrease in salivary flow. 

Serological criterion 

Only three patients had positive RF plus ANA > 1:320 
but negative anti-SSA antibodies; that is, met the ACR 
serological criterion but not the AECG serological criter- 
ion. These three patients fulfilled the AECG criteria; 
they also fulfilled the ACR criteria even without taking 
RF and ANA into account, since they all had abnormal 
OSS and focus score results. 

Ophthalmological criterion 

Agreement between the OSS (ACR set) and Schirmers test 
(AECG set) was very low (k = 0.14). Agreement with the 
salivary gland biopsy was lower for the OSS than for Schir- 
mer s test (k = 0.14 vs. 0.35, respectively). Both the OSS and 
Schirmers test showed poor agreement with anti-SSA/SSB 
positivity (k = 0.21 and k = 0.27 respectively). 

Discussion 

In this study, agreement was only moderate between the 
AECG and ACR criteria sets. The AECG criteria set 
classified more patients as having SS, whereas the ACR 
criteria set seemed to classify patients earlier in the 
course of the disease but included patients who did not 
have SS in the opinion of the physician. These discrep- 
ancies were chiefly ascribable to the absence in the ACR 
set of functional salivary gland testing (such as salivary 
flow measurement) and, above all, to the intrinsic differ- 
ences between the OSS and Schirmers test. 

These two ocular dryness tests had very low agreement. 
According to the latent class analysis method used to de- 
velop the ACR classification criteria [2], the OSS had 
89.7% sensitivity but only 37.8% specificity for SS, whereas 
Schirmer s test had a lower sensitivity of 42.7% but a better 
specificity of 75.1%. In our study, in patients fulfilling nei- 
ther criteria set (and who were therefore unlikely to be di- 
agnosed with SS), an abnormal OSS was more common 
than an abnormal Schirmer s test, suggesting lower specifi- 
city of the OSS. To compare the diagnostic usefulness of 
these two tests, we did not use the physician s diagnosis as 
the reference, since this diagnosis relied chiefly on 
Schirmers test and not on the OSS. Instead, we evaluated 
the agreement of the ocular dryness tests with the focus 



score and anti-SSA/SSB positivity, two major diagnostic 
features of SS [6] that served as external validation criteria. 
Compared with the OSS, Schirmers test showed slightly 
better agreement with the focus score. 

In the recently published study by Rasmussen and 
colleagues, the discordance between AECG and ACR 
criteria was also mostly attributed to the differences be- 
tween the tests assessing the objective ocular component 
[7]. In their study, OSS displayed a poor specificity for 
SS (45 to 51%), which could be partially corrected by in- 
creasing its positivity cutoff value from > 3 to > 4/12. 

A complete ophthalmological evaluation is mandatory 
in patients with suspected SS, in particular to assess eye- 
lid diseases and the differential diagnoses of keratocon- 
junctivitis sicca. However, classification tools should rely 
on the most specific items. Advantages of Schirmer s test 
include ease of use, even at the bedside in any clinical 
ward where trained staff members are available, and 
good performance as a screening tool for SS [8]. 

The serological item of the AECG criteria could prob- 
ably be improved, since roughly 40% of primary SS pa- 
tients do not have anti-SSA/SSB antibodies. However, 
we have shown here that the adjunction of ANA and RF, 
as proposed in the ACR criteria, did not modify the clas- 
sification potential of anti-SSA/SSB alone. Other tests 
should be evaluated when new classification criteria will 
be developed, such as blood B-cell phenotyping [9] or 
other autoantibodies [10]. 

Another intrinsic difference between the AECG cri- 
teria and the ACR criteria is the absence of items asses- 
sing the subjective component of the disease in the 
latter. Indeed, the ACR criteria do not target the general 
population but only patients with suspected SS, who 
complain most of the time of sicca symptoms as in our 
study. Conversely, the AECG criteria may be applied to 
any patient thanks to the inclusion of the symptoms in 
items 1 and 2 (see Table 1). However, the preliminary 
European criteria and then the AECG criteria were not 
designed to be used in the general population, since the 
control patients enrolled in their development study 
were to be selected from those subjects referred to an 
SS expert because of ocular or oral signs and symptoms 
that simulated the clinical manifestations of SS, and for 
whom a complete evaluation was justified in order to es- 
tablish a differential diagnosis' [11]. The precise diagnos- 
tic value of several questionnaires assessing ocular and 
oral symptoms has been carefully evaluated in these pa- 
tients with suspected SS [12]. Such questionnaires may 
be valid tools for SS screening in the general population 
[8], but in our study the subjective symptoms did not 
participate in the discrepancy between AECG and ACR 
criteria, since nearly all patients had sicca complaints. 

An earlier study compared the AECG criteria, the 
ACR criteria and the Japanese classification criteria sets 
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for SS [13]. Taking the physicians diagnosis as the refer- 
ence standard, the AECG and ACR criteria sets had 
78.6% and 77.5% sensitivity, respectively, and 90.4% and 
83.5% specificity. 

However, a direct comparison of the AECG and ACR 
criteria sets in terms of sensitivity and specificity, taking 
the physician s diagnosis as the reference standard, may 
be inherently biased, as physicians probably rely heavily 
on the currently used AECG criteria set to diagnose SS. 
The resulting circular reasoning may overestimate the 
metrological features of the AECG criteria set. 

Classification criteria for systemic diseases such as SS, 
rheumatoid arthritis, systemic lupus erythematosus, sys- 
temic sclerosis and inflammatory myopathies are de- 
signed to improve the homogeneity of populations 
enrolled in clinical studies, in order to allow valid com- 
parisons across studies [14-17]. Since no specific refer- 
ence standard is available for diagnosing these complex 
diseases, classification criteria are often used for diag- 
nostic purposes, despite their limitations [18-20]. 

Whether classification criteria should have a higher 
sensitivity or specificity is a matter of debate. Foremost, 
classification criteria might be used to recruit patients in 
epidemiological studies; for example, to describe the 
whole spectrum of a disease including the mildest forms, 
and to define prognostic factors. For such a study, the 
classification criteria sensitivity should be high, even to 
the detriment of their specificity, since it would not be 
dangerous to include patients for whom the diagnosis is 
not definitely certain. Conversely, in therapeutic trials 
using potentially harmful drugs such as immunosuppres- 
sant or biological therapies, one could not take the risk 
of including patients who do not have the disease. In 
that case, the classification criteria have to be the most 
specific possible. All published classification criteria for 
SS, including preliminary European and AECG criteria, 
have a very high specificity (95 to 100%) but variable 
sensitivity (60 to 95%) [21]. The scientific community 
must achieve the widest consensus for a new classifica- 
tion system, which would display the best combination 
of sensitivity and specificity in order to be used univer- 
sally in both epidemiological and therapeutic studies. To 
achieve this goal, a large international study is war- 
ranted, and the new classification criteria should include 
new diagnostic tools that were validated recently, such 
as salivary gland ultrasonography [4,22-24], 

Conclusions 

In this study, agreement was only moderate between the 
AECG and ACR criteria sets. This discrepancy was mainly 
ascribable to the intrinsic differences between the tests 
assessing the ocular component of the criteria. Our results 
suggest that the ACR criteria may detect early forms of 
disease affecting specific SS subpopulations such as those 



with negative anti-SSA/SSB autoantibodies. On the other 
hand, the AECG criteria seem definitely more specific but 
also more stringent. The existence of two different classi- 
fication criteria sets for SS that select different patient 
populations may cause confusion [25,26]. An international 
study under the auspices of both the ACR and the 
European League Against Rheumatism is warranted to 
develop new universally recognized classification cri- 
teria, which would probably include new procedures 
such as major salivary gland ultrasonography. 
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